GQM++ A Full Life Cycle Framework for the Development and Implementation of Software Metric Programs

One of the more challenging aspects concerning the development of a sound and maturing software process is the determination of an appropriate and relevant supporting measurement program. At one time process and product measurement was a fantasy, something that was undertaken by other organisations. With the advent of process improvement models, metric program development has now gained extensive support. Moreover, the use of frameworks such as the Goal/Question/Metric (GQM) approach has enabled managers to more effectively and confidently select candidate metrics or measures. It is our contention however, that GQM only takes us some way to the development of a feasible program, and that there are other considerations that should be made before embarking on organisation-wide collection schemes. These considerations include in particular an explicit acknowledgment of the costs and benefits of data collection, the specification of potential modelling and analysis methods, and the determination of how any results might be ‘fed back’ into the process. We therefore propose an extension to the GQM framework, which we have called GQM++, to enable the more formal consideration of such issues. Although as yet untested, it is our opinion that explicit specification of these other metric program characteristics will result in the development of more comprehensive, pragmatic and feasible data collection and analysis processes. Introduction Modern software development has become increasingly complex and difficult to manage as the size and functionality of systems have grown and as increasing numbers of competing and co-operative development tools and methodologies have become available. Such circumstances make the use of software metrics even more crucial than in the past. Software metric models can be used to predict, classify or analyze the effects of aspects of the software development process on other aspects. For example, development effort may be predicted based on a set of size and complexity measures, modules may be classified into those suitable for reuse and those that are not, and a particular methodology may be analyzed in terms of its effect on developer productivity as compared to another methodology. Metrics can be used at various stages of the software process – before development as part of project planning, during development to monitor and control, and after development is completed to assess the project’s ‘success’. One major difficulty associated with the adoption of a metrics program is in the determination of the particular metrics that should be collected. All data collection has associated costs, both direct (as in terms of people employed and software purchased) and indirect (as in the costs of interruptions and delays as metrics are analyzed). For this reason, a commercial metrics program should be treated as an investment in the development process, requiring justification for expenditure and providing some quantification of the resulting benefits. For researchers in the metrics field similar concerns arise – data collection still generally incurs costs and the metrics selected must be capable of producing the desired outcome. Organizations are faced with a wide range of software metrics in both the research literature and as promoted by metrics consultants. This can lead to difficulties in the selection of the appropriate metrics to collect for a particular organization with particular goals in mind. Given the uniqueness of each organization, the application of some generic set of metrics is unlikely to be as effective as a more tailored approach. Similarly within a research context, a custom set of metrics, possibly including metrics that have not been used before, will generally provide more meaningful results and at the very least should advance the discipline into new areas. When developing a set of software metrics, whether the purpose is for the actual management of a commercial software development project or as part of a research program, it is important to ensure that the optimal set of metrics is chosen. By optimal it is not just meant that the metrics should be those that would provide the greatest predictive power for the desired purpose. It is also important that the cost of data collection is minimized (or balanced with the benefits to be gained from the accuracy of the model). The possibility for reusing metrics collected for other purposes should also be considered as part of the collection cost. The data collected should of course be useful from the perspective of enabling appropriate models to be developed. Even more important however, is that the metrics should provide information that will enable progress towards some improvement-oriented goal or goals. As can be seen, there are several separate dimensions to the suitability (or optimality) of a given set of metrics for a particular organization or research program. In order to develop a set of metrics that is optimal in this sense it seems useful to work within a framework that will assist with the decomposition of goals into the metrics that should be collected, that will identify the costs and benefits of the program, and ensure that the entire process is indeed feasible. This paper outlines the currently most popular such approach to selecting suites of metrics with some goal in mind, the Goal/Question/Metric (GQM) framework, and attempts to resolve some of its weaknesses through additional stages. Since the GQM framework is already widely used, it is hoped that these incremental changes will be comparatively simple for many organizations and researchers to use where appropriate. The Goal/Question/Metric Framework One method for selecting a suite of metrics to be captured with a goal (or set of goals) in mind is the Goal/Question/Metric (GQM) framework. GQM was proposed by Basili and others (Basili and Weiss 1984; Basili and Rombach 1988) with the aim of providing a systematic approach to the determination of project goals that can be refined into questions that can in turn be answered in a quantifiable manner by certain software metrics (Figure 1, with an example in Figure 2). A goal is usually defined in terms of a particular purpose (e.g. to evaluate, predict, classify) from a certain perspective (e.g. the manager, customer, developer) for a given object (e.g. specification correctness, code modularity), within a particular environment (e.g. the people, tools, methodologies). Question Question Question Metric Metric Metric Metric Metric Metric Goal Figure 1 The GQM Framework Does Size Affect Productivity? What is Current Productivity? Does Team Makeup Affect Productivity? Pages of Documentation Lines of Code Function Points Developer Experience Developer Effort Team Size Improve Productivity Figure 2 An Example of a GQM Framework Thus, the effective application of GQM helps to ensure that the metrics collected will indeed be useful for the purposes of achieving certain goals. It also assists with the conceptual understanding of those goals and metrics by providing a hierarchical framework within which the goal(s) can be broken down into questions, that then in turn determine the actual metrics. GQM has been extended from this basic model by several authors. Extensions include those shown in MacDonell (1993) where the basic framework is extended into a hierarchy of goals, subgoals, domains, subdomains, questions, subquestions, characteristic measures, and finally measures at two separate levels appropriate to his domain of CASE and 4GL effort prediction (Figure 3, with an example adapted from MacDonell in Figure 4). While these extensions of GQM apparently reduce some of the simplicity and ease of use inherent in the original technique, it is felt that as long as these extensions are seen as optional, and therefore only used when necessary, they actually reduce the complexity of the hierarchy by enabling a more consistent and systematic breakdown of the components. For example, by breaking a goal into separate sub-goals the essential differences between metrics needed for each sub-goal can be identified, while still identifying the common measures used for both (Figure 5). This may be important if only one sub-goal is ultimately pursued, and assists with matching the costs and benefits of parts of a metrics program.